Estimating the Rate of Web Page Updates

Estimating the rate of Web page updates helps in improving Web crawler’s scheduling policy. But, most of the Web sources are autonomous and updated independently. Clients like Web crawlers are not aware of when and how often the sources change. Unlike other studies, we model the process of Web page updates as non-homogeneous Poisson process and focus on determining localized rate of updates. Then we discuss various rate estimators, showing experimentally how precise they are. This paper explores two classes of problems. Firstly we estimate localized rate of updates by dividing the given sequence of independent and inconsistent update points into consistent windows. From various experimental comparisons, the proposed Weibull estimator outperforms both Intuitive and Duane plot(another proposed estimator) in 91.5%(96.1%) of the whole windows for synthetic(real Web) datasets. Secondly, we predict future update points based on most recent window, where Duane plot estimator outperforms the rest in terms of repository freshness.